Sorry... I've been working with cloudera support - no real answer yet. Anyway, it's a spark job, and we have proven it can work with 7.1.6 in 2 different clusters, so we have 2 that work and 2 that don't. All running CDP 7.1.6. Here's the perms for the /user/yarn directory on the cluster that doesn't work: varneyg@hdadmgw01mxm1:~$ hdfs dfs -ls -d /user/yarn drwxr-xr-x - yarn yarn 0 2021-08-25 14:08 /user/yarn varneyg@hdadmgw01mxm1:~$ hdfs dfs -ls /user/yarn Found 3 items drwx------ - yarn yarn 0 2021-09-15 10:12 /user/yarn/.staging drwxr-xr-x - yarn yarn 0 2021-09-13 10:32 /user/yarn/google-links drwxr-xr-x - hdfs yarn 0 2021-02-21 13:15 /user/yarn/mapreduce The job runs as a "googleops" user - same code running in both 7.1.4 and 7.1.6 I will attach the part of the job that does the writes, and job job logs.
... View more
