How to redirect the access logs and other logs to external file system?

[#1]

Hi folks,

One of the requirements is to redirect the all kind of logs/ ranger logs/access logs and also other component's logs to external file system. Probably on NFS and not on HDFS or in the DB. Does HDP provide out of the box solution for this?

One workaround I can think of using Flume but wanted to know other approaches?

Regards,

DP

[#2 · Accepted solution]

@Darpan Patel

1) https://github.com/abajwa-hw/logsearch-service unofficial

2) You can use NFS to copy logs

3) Ranger and Solr works together for auditing

[#3]

@Darpan Patel flume is one directional, you can't push data out of hdfs to fs. We have Apache NiFi that you can leverage for moving logs to and from hdfs.