Skip to main content
hadoclc
hadoclc
Gold ⭐️
August 27, 2021
Solved

Job fails after installing CDP 7.1.6

We have a job that was running fine in 7.1.4, but is now failing after we went to 7.1.6.  It fails with permission issues writing to a directory in /user/yarn.  The job runs as a different user, and we are wondering why it started writing what looks like temporary output to a directory under /user/yarn, that the user running job doesn't have access to?

hadoclc
hadoclcAnswer
Gold ⭐️
September 24, 2021

As it turns out, it seems like it was a combination of things that caused this job to fail. First, we installed a minor java version update: went from jdk8u242-b08 to openjdk-8u292-b10. Second, the developer changed the way the files were being written from asynchronous to synchronous. They were using the CompletableFuture.runAsync class before, and took that out to use just normal writes.

View in thread

5 replies

Shelton
Shelton
Gold ⭐️⭐️⭐️
August 28, 2021

@hadoclc 
Can you share more details? What job spark/hive?
Can you share some information about your environment and the code submitted that fails?
What is the permission of /user/yarn
Who and how was the job executed in 7.1.4? Is the same user running the job in 7.1.6?
Please share the logs?

Translate
hadoclc
hadoclcAuthor
Gold ⭐️
September 15, 2021

logs and code attached

Translate
ChethanYM
Gold ⭐️
ChethanYM
Gold ⭐️
September 16, 2021

Hi,

 

As per the application log it seems to be the issue with permissions, Have you tried to run with other users? or have you tried to give access to this user to access the path? 

Check the permissions with other users who have access to this path and try to apply the permission to "googleops" user.

 

{"Event":"SparkListenerTaskEnd","Stage ID":11,"Stage Attempt ID":0,"Task Type":"ResultTask","Task End Reason":{"Reason":"ExceptionFailure","Class Name":"org.apache.hadoop.security.AccessControlException","Description":"Permission denied: user=googleops, access=WRITE, inode=\"/user/yarn/google-links/google-book-feed/_temporary/0\":yarn:yarn:drwxr-xr-x\n\tat org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:504)\n\tat org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:336)\n\tat org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:242)\n\tat

 

Thank you,

Chethan YM

Translate
hadoclc
hadoclcAuthorAnswer
Gold ⭐️
September 24, 2021

As it turns out, it seems like it was a combination of things that caused this job to fail. First, we installed a minor java version update: went from jdk8u242-b08 to openjdk-8u292-b10. Second, the developer changed the way the files were being written from asynchronous to synchronous. They were using the CompletableFuture.runAsync class before, and took that out to use just normal writes.

Translate
Terms & ConditionsAccessibility statement