Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Housekeeping of files under hdfs dfs command permission issue

avatar
New Contributor

I'm trying to housekeep our hdfs filesystem and found alot of very old files that are owned by spark user. I don't remember creating a spark user that would generate these files, hence when I invoke an hdfs dfs -rm -R files...it's giving me a file ownership/permission error. 

Is there any work around I can bypass this? just like sudo in unix? Thank you.

 

1 ACCEPTED SOLUTION

avatar
Super Collaborator

Hello @Grumash ,

 

I believe user=cdp_svc_fc_03 is the spark user which no longer exists. So when you are trying to move the file into the trash folder in the home dir, its failing to create the home dir. You need to create the home dir as the super user (hdfs), then chown it to cdp_svc_fc_03, then it should work.

View solution in original post

2 REPLIES 2

avatar
New Contributor

rm: Failed to move to trash: hdfs://aascmuatns1/data/farecard-batch-mgmt04/filelocations/part-r-00000-fffc9580-c310-44c2-9090-73ecf3133133: Permission denied: user=cdp_svc_fc_03, access=WRITE, inode="/user":hdfs:supergroup:drwxr-xr-x

 

The above the error I'm getting, which I believe is expected.

avatar
Super Collaborator

Hello @Grumash ,

 

I believe user=cdp_svc_fc_03 is the spark user which no longer exists. So when you are trying to move the file into the trash folder in the home dir, its failing to create the home dir. You need to create the home dir as the super user (hdfs), then chown it to cdp_svc_fc_03, then it should work.