question HDFS : Check activity HDFS in Archives of Support Questions (Read Only)

HDFS : Check activity HDFS

maykiwogno — Wed, 05 Oct 2016 15:52:37 GMT

hello all,

i must make an audit about the activity of HDFS, for example who was write on hdfs, who was copy from hdfs ?

It is possible to see that some where ?

regards

Re: HDFS : Check activity HDFS

KuldeepK — Wed, 05 Oct 2016 16:25:06 GMT

@mayki wogno

If you have Ranger configured then its very easy, just enable HDFS plugin and ranger will do this for you.

More information on Ranger - http://hortonworks.com/apache/ranger/

Or if you don't have Ranger then you can check this by referring to hdfs audit logs. ( /var/log/hadoop/hdfs/hdfs-audit.log)

Hope this information helps.

Re: HDFS : Check activity HDFS

maykiwogno — Wed, 05 Oct 2016 16:44:26 GMT

@Kuldeep Kulkarni : thanks, i'll check hdfs-audit.log but now my question how i can know what's command/job is behind this log

for example :

hdfs-audit.log:2016-10-05 03:00:17,303 INFO FSNamesystem.audit: allowed=true ugi=zazi (auth:TOKEN) via oozie/master003@fma.com (auth:TOKEN) ip=/10.xx.224.93 cmd=open src=/user/prote/private/flume/20161004/FlumeData.428492.gz dst=null perm=null proto=rpc callerContext=CLI

Re: HDFS : Check activity HDFS

sshimpi — Wed, 05 Oct 2016 16:52:56 GMT

@mayki wogno

Can you check the same from Ranger UI->Access->Audit, which will display the audit in more formatted. You can pass the screenshot here to get more details.

Ranger UI - http://<ranger_host>:6080

Re: HDFS : Check activity HDFS

maykiwogno — Wed, 05 Oct 2016 16:58:35 GMT

I don't have RANGER

Re: HDFS : Check activity HDFS

KuldeepK — Wed, 05 Oct 2016 18:15:59 GMT

@mayki wogno

Unfortunately, you cann't get job related information however you can guess the job from above information.

Username - zazi

Service principal - oozie/master003@fma.com

Host from where file was accessed - 10.xx.224.9

Please accept the answer if this was helpful.