Our Community is getting an upgrade! To get everything ready for the relaunch, we’ll be placing the site in read-only mode starting September 21st.
We really appreciate your understanding while we get things set up behind the scenes. Catch up on all the exciting details about the move here.
Need help or have questions? Drop us a line at [email protected]
Created 01-18-2017 08:18 AM
how to track / audit the delete operations like rm - rf performed on the HDFS files & directories?
Created 01-18-2017 08:46 AM
Hello @hardik desai
You can get similar information from the /var/log/hadoop/hdfs/hdfs-audit.log log file. However rm would show up as a rename command. ls would show up as listStatus, etc. Here I create a file /tmp/deleteme.txt, upload it to HDFS, list it parent directory, and delete the file.
[hdfs@jyoung-hdp234-1 ~]$ echo "delete me" >> /tmp/deleteme.txt [hdfs@jyoung-hdp234-1 ~]$ hdfs dfs -put /tmp/deleteme.txt /tmp/ [hdfs@jyoung-hdp234-1 ~]$ hdfs dfs -ls /tmp Found 13 items drwx------ - ambari-qa hdfs 0 2016-12-01 04:19 /tmp/ambari-qa drwxrwxrwx - oozie hdfs 0 2016-12-19 08:32 /tmp/crime -rw-r--r-- 3 hdfs hdfs 10 2017-01-18 08:29 /tmp/deleteme.txt drwxr-xr-x - hdfs hdfs 0 2016-12-01 04:15 /tmp/entity-file-history drwx-wx-wx - ambari-qa hdfs 0 2016-12-11 18:36 /tmp/hive -rwxr-xr-x 3 hdfs hdfs 1616 2016-12-01 04:16 /tmp/id1aacdf51_date160116 -rwxr-xr-x 3 hdfs hdfs 1616 2016-12-01 05:19 /tmp/id1aacdf51_date190116 -rwxr-xr-x 3 ambari-qa hdfs 1616 2016-12-01 04:21 /tmp/idtest.ambari-qa.1480566109.56.in -rwxr-xr-x 3 ambari-qa hdfs 957 2016-12-01 04:21 /tmp/idtest.ambari-qa.1480566109.56.pig -rwxr-xr-x 3 ambari-qa hdfs 1616 2016-12-01 05:23 /tmp/idtest.ambari-qa.1480569805.86.in -rwxr-xr-x 3 ambari-qa hdfs 957 2016-12-01 05:23 /tmp/idtest.ambari-qa.1480569805.86.pig drwxr-xr-x - ambari-qa hdfs 0 2016-12-01 04:19 /tmp/tezsmokeinput drwxr-xr-x - ambari-qa hdfs 0 2016-12-01 05:21 /tmp/tezsmokeoutput [hdfs@jyoung-hdp234-1 ~]$ hdfs dfs -rm /tmp/deleteme.txt 17/01/18 08:29:46 INFO fs.TrashPolicyDefault: Namenode trash configuration: Deletion interval = 360 minutes, Emptier interval = 0 minutes. Moved: 'hdfs://jyoung-hdp234-1.openstacklocal:8020/tmp/deleteme.txt' to trash at: hdfs://jyoung-hdp234-1.openstacklocal:8020/user/hdfs/.Trash/Current
You can see these operations in my hdfs-audit.log file below:
[root@jyoung-hdp234-1 hdfs]# grep "CLI" /var/log/hadoop/hdfs/hdfs-audit.log 2017-01-18 08:28:29,254 INFO FSNamesystem.audit: allowed=true [email protected] (auth:KERBEROS) ip=/172.26.81.223 cmd=getfileinfo src=/tmp dst=null perm=null proto=rpc callerContext=CLI 2017-01-18 08:29:13,847 INFO FSNamesystem.audit: allowed=true [email protected] (auth:KERBEROS) ip=/172.26.81.223 cmd=getfileinfo src=/tmp dst=null perm=null proto=rpc callerContext=CLI 2017-01-18 08:29:13,942 INFO FSNamesystem.audit: allowed=true [email protected] (auth:KERBEROS) ip=/172.26.81.223 cmd=getfileinfo src=/tmp/deleteme.txt dst=null perm=null proto=rpc callerContext=CLI 2017-01-18 08:29:13,947 INFO FSNamesystem.audit: allowed=true [email protected] (auth:KERBEROS) ip=/172.26.81.223 cmd=getfileinfo src=/tmp/deleteme.txt._COPYING_ dst=null perm=null proto=rpc callerContext=CLI 2017-01-18 08:29:13,990 INFO FSNamesystem.audit: allowed=true [email protected] (auth:KERBEROS) ip=/172.26.81.223 cmd=create src=/tmp/deleteme.txt._COPYING_ dst=null perm=hdfs:hdfs:rw-r--r-- proto=rpc callerContext=CLI 2017-01-18 08:29:14,022 INFO FSNamesystem.audit: allowed=true [email protected] (auth:KERBEROS) ip=/172.26.81.223 cmd=getfileinfo src=/tmp/deleteme.txt._COPYING_ dst=null perm=null proto=rpc callerContext=CLI 2017-01-18 08:29:14,240 INFO FSNamesystem.audit: allowed=true [email protected] (auth:KERBEROS) ip=/172.26.81.223 cmd=rename src=/tmp/deleteme.txt._COPYING_ dst=/tmp/deleteme.txt perm=hdfs:hdfs:rw-r--r-- proto=rpc callerContext=CLI 2017-01-18 08:29:27,129 INFO FSNamesystem.audit: allowed=true [email protected] (auth:KERBEROS) ip=/172.26.81.223 cmd=getfileinfo src=/tmp dst=null perm=null proto=rpc callerContext=CLI 2017-01-18 08:29:27,247 INFO FSNamesystem.audit: allowed=true [email protected] (auth:KERBEROS) ip=/172.26.81.223 cmd=listStatus src=/tmp dst=null perm=null proto=rpc callerContext=CLI 2017-01-18 08:29:27,466 INFO FSNamesystem.audit: allowed=true ugi=hbase/[email protected] (auth:KERBEROS) ip=/172.26.81.225 cmd=listStatus src=/apps/hbase/data/oldWALs dst=null perm=null proto=rpc callerContext=CLI 2017-01-18 08:29:27,469 INFO FSNamesystem.audit: allowed=true ugi=hbase/[email protected] (auth:KERBEROS) ip=/172.26.81.225 cmd=listStatus src=/apps/hbase/data/archive dst=null perm=null proto=rpc callerContext=CLI 2017-01-18 08:29:46,457 INFO FSNamesystem.audit: allowed=true [email protected] (auth:KERBEROS) ip=/172.26.81.223 cmd=getfileinfo src=/tmp/deleteme.txt dst=null perm=null proto=rpc callerContext=CLI 2017-01-18 08:29:46,552 INFO FSNamesystem.audit: allowed=true [email protected] (auth:KERBEROS) ip=/172.26.81.223 cmd=getfileinfo src=/tmp/deleteme.txt dst=null perm=null proto=rpc callerContext=CLI 2017-01-18 08:29:46,570 INFO FSNamesystem.audit: allowed=true [email protected] (auth:KERBEROS) ip=/172.26.81.223 cmd=getfileinfo src=/tmp/deleteme.txt dst=null perm=null proto=rpc callerContext=CLI 2017-01-18 08:29:46,585 INFO FSNamesystem.audit: allowed=true [email protected] (auth:KERBEROS) ip=/172.26.81.223 cmd=mkdirs src=/user/hdfs/.Trash/Current/tmp dst=null perm=hdfs:hdfs:rwx------ proto=rpc callerContext=CLI 2017-01-18 08:29:46,590 INFO FSNamesystem.audit: allowed=true [email protected] (auth:KERBEROS) ip=/172.26.81.223 cmd=getfileinfo src=/user/hdfs/.Trash/Current/tmp/deleteme.txt dst=null perm=null proto=rpc callerContext=CLI 2017-01-18 08:29:46,595 INFO FSNamesystem.audit: allowed=true [email protected] (auth:KERBEROS) ip=/172.26.81.223 cmd=rename src=/tmp/deleteme.txt dst=/user/hdfs/.Trash/Current/tmp/deleteme.txt perm=hdfs:hdfs:rw-r--r-- proto=rpc callerContext=CLI
Created 01-18-2017 08:46 AM
Hello @hardik desai
You can get similar information from the /var/log/hadoop/hdfs/hdfs-audit.log log file. However rm would show up as a rename command. ls would show up as listStatus, etc. Here I create a file /tmp/deleteme.txt, upload it to HDFS, list it parent directory, and delete the file.
[hdfs@jyoung-hdp234-1 ~]$ echo "delete me" >> /tmp/deleteme.txt [hdfs@jyoung-hdp234-1 ~]$ hdfs dfs -put /tmp/deleteme.txt /tmp/ [hdfs@jyoung-hdp234-1 ~]$ hdfs dfs -ls /tmp Found 13 items drwx------ - ambari-qa hdfs 0 2016-12-01 04:19 /tmp/ambari-qa drwxrwxrwx - oozie hdfs 0 2016-12-19 08:32 /tmp/crime -rw-r--r-- 3 hdfs hdfs 10 2017-01-18 08:29 /tmp/deleteme.txt drwxr-xr-x - hdfs hdfs 0 2016-12-01 04:15 /tmp/entity-file-history drwx-wx-wx - ambari-qa hdfs 0 2016-12-11 18:36 /tmp/hive -rwxr-xr-x 3 hdfs hdfs 1616 2016-12-01 04:16 /tmp/id1aacdf51_date160116 -rwxr-xr-x 3 hdfs hdfs 1616 2016-12-01 05:19 /tmp/id1aacdf51_date190116 -rwxr-xr-x 3 ambari-qa hdfs 1616 2016-12-01 04:21 /tmp/idtest.ambari-qa.1480566109.56.in -rwxr-xr-x 3 ambari-qa hdfs 957 2016-12-01 04:21 /tmp/idtest.ambari-qa.1480566109.56.pig -rwxr-xr-x 3 ambari-qa hdfs 1616 2016-12-01 05:23 /tmp/idtest.ambari-qa.1480569805.86.in -rwxr-xr-x 3 ambari-qa hdfs 957 2016-12-01 05:23 /tmp/idtest.ambari-qa.1480569805.86.pig drwxr-xr-x - ambari-qa hdfs 0 2016-12-01 04:19 /tmp/tezsmokeinput drwxr-xr-x - ambari-qa hdfs 0 2016-12-01 05:21 /tmp/tezsmokeoutput [hdfs@jyoung-hdp234-1 ~]$ hdfs dfs -rm /tmp/deleteme.txt 17/01/18 08:29:46 INFO fs.TrashPolicyDefault: Namenode trash configuration: Deletion interval = 360 minutes, Emptier interval = 0 minutes. Moved: 'hdfs://jyoung-hdp234-1.openstacklocal:8020/tmp/deleteme.txt' to trash at: hdfs://jyoung-hdp234-1.openstacklocal:8020/user/hdfs/.Trash/Current
You can see these operations in my hdfs-audit.log file below:
[root@jyoung-hdp234-1 hdfs]# grep "CLI" /var/log/hadoop/hdfs/hdfs-audit.log 2017-01-18 08:28:29,254 INFO FSNamesystem.audit: allowed=true [email protected] (auth:KERBEROS) ip=/172.26.81.223 cmd=getfileinfo src=/tmp dst=null perm=null proto=rpc callerContext=CLI 2017-01-18 08:29:13,847 INFO FSNamesystem.audit: allowed=true [email protected] (auth:KERBEROS) ip=/172.26.81.223 cmd=getfileinfo src=/tmp dst=null perm=null proto=rpc callerContext=CLI 2017-01-18 08:29:13,942 INFO FSNamesystem.audit: allowed=true [email protected] (auth:KERBEROS) ip=/172.26.81.223 cmd=getfileinfo src=/tmp/deleteme.txt dst=null perm=null proto=rpc callerContext=CLI 2017-01-18 08:29:13,947 INFO FSNamesystem.audit: allowed=true [email protected] (auth:KERBEROS) ip=/172.26.81.223 cmd=getfileinfo src=/tmp/deleteme.txt._COPYING_ dst=null perm=null proto=rpc callerContext=CLI 2017-01-18 08:29:13,990 INFO FSNamesystem.audit: allowed=true [email protected] (auth:KERBEROS) ip=/172.26.81.223 cmd=create src=/tmp/deleteme.txt._COPYING_ dst=null perm=hdfs:hdfs:rw-r--r-- proto=rpc callerContext=CLI 2017-01-18 08:29:14,022 INFO FSNamesystem.audit: allowed=true [email protected] (auth:KERBEROS) ip=/172.26.81.223 cmd=getfileinfo src=/tmp/deleteme.txt._COPYING_ dst=null perm=null proto=rpc callerContext=CLI 2017-01-18 08:29:14,240 INFO FSNamesystem.audit: allowed=true [email protected] (auth:KERBEROS) ip=/172.26.81.223 cmd=rename src=/tmp/deleteme.txt._COPYING_ dst=/tmp/deleteme.txt perm=hdfs:hdfs:rw-r--r-- proto=rpc callerContext=CLI 2017-01-18 08:29:27,129 INFO FSNamesystem.audit: allowed=true [email protected] (auth:KERBEROS) ip=/172.26.81.223 cmd=getfileinfo src=/tmp dst=null perm=null proto=rpc callerContext=CLI 2017-01-18 08:29:27,247 INFO FSNamesystem.audit: allowed=true [email protected] (auth:KERBEROS) ip=/172.26.81.223 cmd=listStatus src=/tmp dst=null perm=null proto=rpc callerContext=CLI 2017-01-18 08:29:27,466 INFO FSNamesystem.audit: allowed=true ugi=hbase/[email protected] (auth:KERBEROS) ip=/172.26.81.225 cmd=listStatus src=/apps/hbase/data/oldWALs dst=null perm=null proto=rpc callerContext=CLI 2017-01-18 08:29:27,469 INFO FSNamesystem.audit: allowed=true ugi=hbase/[email protected] (auth:KERBEROS) ip=/172.26.81.225 cmd=listStatus src=/apps/hbase/data/archive dst=null perm=null proto=rpc callerContext=CLI 2017-01-18 08:29:46,457 INFO FSNamesystem.audit: allowed=true [email protected] (auth:KERBEROS) ip=/172.26.81.223 cmd=getfileinfo src=/tmp/deleteme.txt dst=null perm=null proto=rpc callerContext=CLI 2017-01-18 08:29:46,552 INFO FSNamesystem.audit: allowed=true [email protected] (auth:KERBEROS) ip=/172.26.81.223 cmd=getfileinfo src=/tmp/deleteme.txt dst=null perm=null proto=rpc callerContext=CLI 2017-01-18 08:29:46,570 INFO FSNamesystem.audit: allowed=true [email protected] (auth:KERBEROS) ip=/172.26.81.223 cmd=getfileinfo src=/tmp/deleteme.txt dst=null perm=null proto=rpc callerContext=CLI 2017-01-18 08:29:46,585 INFO FSNamesystem.audit: allowed=true [email protected] (auth:KERBEROS) ip=/172.26.81.223 cmd=mkdirs src=/user/hdfs/.Trash/Current/tmp dst=null perm=hdfs:hdfs:rwx------ proto=rpc callerContext=CLI 2017-01-18 08:29:46,590 INFO FSNamesystem.audit: allowed=true [email protected] (auth:KERBEROS) ip=/172.26.81.223 cmd=getfileinfo src=/user/hdfs/.Trash/Current/tmp/deleteme.txt dst=null perm=null proto=rpc callerContext=CLI 2017-01-18 08:29:46,595 INFO FSNamesystem.audit: allowed=true [email protected] (auth:KERBEROS) ip=/172.26.81.223 cmd=rename src=/tmp/deleteme.txt dst=/user/hdfs/.Trash/Current/tmp/deleteme.txt perm=hdfs:hdfs:rw-r--r-- proto=rpc callerContext=CLI
Created 01-18-2017 09:24 AM
@jyoung, i can relate to your answer and descreption...thanks!