Skip to main content
Grumash
Silver ⭐️
Jun 22, 2022 9:11 AM UTC
Solved

Housekeeping of files under hdfs dfs command permission issue

I'm trying to housekeep our hdfs filesystem and found alot of very old files that are owned by spark user. I don't remember creating a spark user that would generate these files, hence when I invoke an hdfs dfs -rm -R files...it's giving me a file ownership/permission error. 

Is there any work around I can bypass this? just like sudo in unix? Thank you.

 

rki_
rki_Answer
Gold ⭐️
Jun 30, 2022 10:41 AM UTC

Hello @Grumash ,

 

I believe user=cdp_svc_fc_03 is the spark user which no longer exists. So when you are trying to move the file into the trash folder in the home dir, its failing to create the home dir. You need to create the home dir as the super user (hdfs), then chown it to cdp_svc_fc_03, then it should work.

2 replies

Grumash
GrumashAuthor
Silver ⭐️
Jun 22, 2022 9:18 AM UTC

rm: Failed to move to trash: hdfs://aascmuatns1/data/farecard-batch-mgmt04/filelocations/part-r-00000-fffc9580-c310-44c2-9090-73ecf3133133: Permission denied: user=cdp_svc_fc_03, access=WRITE, inode="/user":hdfs:supergroup:drwxr-xr-x

 

The above the error I'm getting, which I believe is expected.

Translate
rki_
Gold ⭐️
rki_Answer
Gold ⭐️
Jun 30, 2022 10:41 AM UTC

Hello @Grumash ,

 

I believe user=cdp_svc_fc_03 is the spark user which no longer exists. So when you are trying to move the file into the trash folder in the home dir, its failing to create the home dir. You need to create the home dir as the super user (hdfs), then chown it to cdp_svc_fc_03, then it should work.

Translate