HDFS : Check activity HDFS

[#1]

hello all,

i must make an audit about the activity of HDFS, for example who was write on hdfs, who was copy from hdfs ?

It is possible to see that some where ?

regards

[#2 · Accepted solution]

@mayki wogno

If you have Ranger configured then its very easy, just enable HDFS plugin and ranger will do this for you.

More information on Ranger - http://hortonworks.com/apache/ranger/

Or if you don't have Ranger then you can check this by referring to hdfs audit logs. ( /var/log/hadoop/hdfs/hdfs-audit.log)

Hope this information helps.

[#3]

@Kuldeep Kulkarni : thanks, i'll check hdfs-audit.log but now my question how i can know what's command/job is behind this log

for example :

hdfs-audit.log:2016-10-05 03:00:17,303 INFO FSNamesystem.audit: allowed=true ugi=zazi (auth:TOKEN) via oozie/[email protected] (auth:TOKEN) ip=/10.xx.224.93 cmd=open src=/user/prote/private/flume/20161004/FlumeData.428492.gz dst=null perm=null proto=rpc callerContext=CLI

[#4]

@mayki wogno

Unfortunately, you cann't get job related information however you can guess the job from above information.

Username - zazi

Service principal - oozie/[email protected]

Host from where file was accessed - 10.xx.224.9

Please accept the answer if this was helpful.

[#5]

@mayki wogno

Can you check the same from Ranger UI->Access->Audit, which will display the audit in more formatted. You can pass the screenshot here to get more details.

Ranger UI - http://<ranger_host>:6080

[#6]

I don't have RANGER