Skip to main content
hadoclc
hadoclc
Gold ⭐️
August 27, 2021
Solved

Job fails after installing CDP 7.1.6

We have a job that was running fine in 7.1.4, but is now failing after we went to 7.1.6.  It fails with permission issues writing to a directory in /user/yarn.  The job runs as a different user, and we are wondering why it started writing what looks like temporary output to a directory under /user/yarn, that the user running job doesn't have access to?

hadoclc
hadoclcAnswer
Gold ⭐️
September 24, 2021

As it turns out, it seems like it was a combination of things that caused this job to fail. First, we installed a minor java version update: went from jdk8u242-b08 to openjdk-8u292-b10. Second, the developer changed the way the files were being written from asynchronous to synchronous. They were using the CompletableFuture.runAsync class before, and took that out to use just normal writes.

View in thread

5 replies

Shelton
Shelton
Gold ⭐️⭐️⭐️
August 28, 2021

@hadoclc 
Can you share more details? What job spark/hive?
Can you share some information about your environment and the code submitted that fails?
What is the permission of /user/yarn
Who and how was the job executed in 7.1.4? Is the same user running the job in 7.1.6?
Please share the logs?

Translate
hadoclc
hadoclcAuthor
Gold ⭐️
September 15, 2021

Sorry...   I've been working with cloudera support - no real answer yet.  Anyway, it's a spark job, and we have proven it can work with 7.1.6 in 2 different clusters, so we have 2 that work and 2 that don't.  All running CDP 7.1.6.  Here's the perms for the /user/yarn directory on the cluster that doesn't work:

varneyg@hdadmgw01mxm1:~$ hdfs dfs -ls -d /user/yarn
drwxr-xr-x - yarn yarn 0 2021-08-25 14:08 /user/yarn
varneyg@hdadmgw01mxm1:~$ hdfs dfs -ls /user/yarn
Found 3 items
drwx------ - yarn yarn 0 2021-09-15 10:12 /user/yarn/.staging
drwxr-xr-x - yarn yarn 0 2021-09-13 10:32 /user/yarn/google-links
drwxr-xr-x - hdfs yarn 0 2021-02-21 13:15 /user/yarn/mapreduce

 

The job runs as a "googleops" user - same code running in both 7.1.4 and 7.1.6  I will attach the part of the job that does the writes, and job job logs.

Translate
hadoclc
hadoclcAuthorAnswer
Gold ⭐️
September 24, 2021

As it turns out, it seems like it was a combination of things that caused this job to fail. First, we installed a minor java version update: went from jdk8u242-b08 to openjdk-8u292-b10. Second, the developer changed the way the files were being written from asynchronous to synchronous. They were using the CompletableFuture.runAsync class before, and took that out to use just normal writes.

Translate
Terms & ConditionsAccessibility statement