Created 04-02-2019 12:41 PM
Hello,
I have a scenario with a Hadoop cluster installed with HDP2.6.5 and a Kafka cluster installed with HDF 3.3.0 with Ranger Service configured.
I want to store the Ranger Audit logs in HDFS so I setup in kafka the property xasecure.audit.destination.hdfs.dir pointing to the HDFS directory.
Case one: when using the namenode in the URI the logs are stored in HDFS successfully (xasecure.audit.destination.hdfs.dir=hdfs://namenode_FQDN>:8020/ranger/audit)
Case two: Using a haproxy, since i have namenode HA enabled and want to point always to the active NN, i get the following error
2019-04-02 12:00:13,841 ERROR [kafka.async.summary.multi_dest.batch_kafka.async.summary.multi_dest.batch.hdfs_destWriter] org.apache.ranger.audit.provider.BaseAuditHandler (BaseAuditHandler.java:329) - Error writing to log file. java.io.IOException: DestHost:destPort <ha_proxy_hostname>:8085 , LocalHost:localPort <kafka_broker_hostname>/10.212.164.50:0. Failed on local exception: java.io.IOException: org.apache.hadoop.ipc.RpcException: RPC response exceeds maximum data length
Is there any extra config to be set?
Thanks
Created 09-19-2019 01:32 AM
Hi, this is what i did:
Kafka→ Configs→ Advanced ranger-kafka-audit and add the dfs destination dir
(if you have NameNode HA, you need to add to each kafka broker the hdfs-site.xml that has the nameservice property, so the audit logs should always hit the active namenode)
For example if you have defined the fs.defaultFS=nameservice you will add something like
xasecure.audit.destination.hdfs.dir=hdfs://nameservice/ranger/audit
Created 04-03-2019 04:42 PM
Hi,
Could you please share hdf instalaltion document. I want to install HDF installation on my personal computer.
Created 04-09-2019 07:22 AM
I followed the steps in this link https://docs.hortonworks.com/HDPDocuments/HDF3/HDF-3.2.0/installing-hdf/content/install-ambari.html
Created 04-09-2019 07:20 AM
Anyone can help on this topic?
Created 04-26-2019 12:57 PM
Found how to proceed and now i can store the logs in HDFS
Created on 09-17-2019 01:40 AM - edited 09-17-2019 01:44 AM
Could you please share how did you proceed ?
Thank you
Created 09-19-2019 01:32 AM
Hi, this is what i did:
Kafka→ Configs→ Advanced ranger-kafka-audit and add the dfs destination dir
(if you have NameNode HA, you need to add to each kafka broker the hdfs-site.xml that has the nameservice property, so the audit logs should always hit the active namenode)
For example if you have defined the fs.defaultFS=nameservice you will add something like
xasecure.audit.destination.hdfs.dir=hdfs://nameservice/ranger/audit
Created 09-19-2019 01:36 AM
Sorry, forgot to add the port
the correct way will be
hdfs://nameservice:8020/ranger/audit
Created 10-04-2019 05:01 AM
It works only if it's the same KDC, need cross trust (realm) in my case.
Thank you.
ps : I didn't got the notification too
Regards
Created 09-17-2019 02:11 PM
@psilvarochagome
In this community, we share knowledge to advance the Cloudera community and don't get cash for that! though some are real production issues, having said that it's unfortunate people like you got a solution to a problem being faced by a member and don't want to share as requested by @slim_abderrahim
It's very unfortunate I hope member see this and tag you ... .........we open-source as opposed to proprietary code. 🙂
Happy hadooping